A data processing method and related device

Abstract

The embodiment of the present application discloses a data processing method and related devices. The decoder of the language model includes a multi-layer attention layer, which includes a first attention layer and a second attention layer that are adjacent to each other in the decoding order. At least two attention mechanisms are set in the multi-layer attention layer, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism. For the corpus to be processed, the language model can be used to obtain the language processing result corresponding to the corpus to be processed based on the input data determined by the corpus to be processed. Since the language model can adopt at least two different attention mechanisms, the attention distribution or the focus of attention of different attention mechanisms for the same corpus is different. By integrating different attention distributions, the language model can pay attention to more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method and related devices. Background Art

[0002] The language model can enable the machine to realize the ability of natural language processing (NLP), thereby providing users with various language-related services, such as machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, Chinese OCR, etc.

[0003] In order to improve service effects, high-quality language models are needed. Related technologies often use methods such as increasing the number of model parameters, such as increasing the number of model layers.

[0004] However, increasing the number of model parameters will not only increase the complexity of the model, but also have limited effect on improving the quality of the model. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a data processing method and related devices. By integrating different attention distributions, the language model can focus on more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] On the one hand, an embodiment of the present application provides a data processing method, the method comprising:

[0008] Determine a decoder of the language model includes multiple attention layers, wherein the multiple attention layers connected sequentially include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order;

[0009] At least two attention mechanisms are set in the multi-layer attention layer, and the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism;

[0010] Determining input data of the language model according to the corpus to be processed;

[0011] Based on the input data, determining a language processing result of the corpus to be processed by using the language model;

[0012] In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the process includes:

[0013] Determine first sub-input data of the first attention layer based on the input data;

[0014] Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism;

[0015] The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

[0016] On the other hand, an embodiment of the present application provides a data processing device, the device comprising a first determining unit, a setting unit, a second determining unit and a third determining unit:

[0017] The first determining unit is used to determine the multiple attention layers included in the decoder of the language model, the multiple attention layers connected in sequence include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order;

[0018] The setting unit is used to set at least two attention mechanisms in the multi-layer attention layer, the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism;

[0019] The second determining unit is used to determine the input data of the language model according to the corpus to be processed;

[0020] The third determining unit is used to determine the language processing result of the corpus to be processed through the language model based on the input data;

[0021] In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the third determining unit is further used to:

[0022] Determine first sub-input data of the first attention layer based on the input data;

[0023] Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism;

[0024] The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

[0025] On the other hand, an embodiment of the present application discloses a computer device, wherein the computer device includes a processor and a memory:

[0026] The memory is used to store program codes and transmit the program codes to the processor;

[0027] The processor is used to execute the data processing method described in the above aspects according to the instructions in the program code.

[0028] On the other hand, an embodiment of the present application discloses a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and the computer program is used to execute the data processing method described in the above aspects.

[0029] It can be seen from the above technical solution that the decoder for determining the language model includes multiple attention layers, including the first attention layer and the second attention layer that are adjacent to each other in the decoding order. At least two attention mechanisms are set in the multiple attention layers, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism, so as to construct a language model that can be decoded using at least two attention mechanisms. For the corpus to be processed, the language model can be used to obtain the language processing result corresponding to the corpus to be processed based on the input data determined by the corpus to be processed. During the processing, the first attention layer of the decoder of the language model will obtain the first sub-input data determined according to the input data, and the first attention layer can decode it based on the first attention mechanism, and the first sub-processing result obtained reflects the first attention distribution of the first attention mechanism. On the basis of the first sub-processing result, the second attention layer continues to perform decoding processing based on the second attention mechanism, and the second sub-processing result obtained reflects the second attention distribution of the second attention mechanism under the first attention distribution. Therefore, when processing the corpus to be processed, the language model can adopt at least two different attention mechanisms. Different attention mechanisms have different attention distributions or focuses on the same corpus. By integrating different attention distributions, the language model can pay attention to more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A schematic diagram of a language model structure in a translation scenario provided in an embodiment of the present application;

[0032] Figure 2 A method flow chart of a data processing method provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the model structure using a multi-head self-attention mechanism;

[0034] Figure 4 A schematic diagram of a translation scenario process using a language model provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of a translation scenario interaction provided in an embodiment of the present application;

[0036] Figure 6 A model structure diagram of a multi-head self-attention mechanism provided in an embodiment of the present application;

[0037] Figure 7 A model structure diagram of an average attention mechanism provided in an embodiment of the present application;

[0038] Figure 8 A model structure diagram of a multi-head self-attention mechanism priority provided in an embodiment of the present application;

[0039] Fig. 9 A model structure diagram of a dual attention mechanism provided in an embodiment of the present application;

[0040] Fig.10 A model structure diagram of an average attention mechanism priority provided in an embodiment of the present application;

[0041] Fig.11 A device structure diagram of a data processing device provided in an embodiment of the present application;

[0042] Fig.12 A structural diagram of a terminal device provided in an embodiment of the present application;

[0043] Fig.13 A structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The embodiments of the present application are described below in conjunction with the accompanying drawings.

[0045] In related technologies, the means to improve the quality of language models mainly involves increasing the model scale to improve model parameters. This method not only significantly reduces the efficiency of the model, but also has limited improvement on the quality of the model, which is not conducive to the development of language models.

[0046] To this end, an embodiment of the present application provides a data processing method that can focus on more data that requires attention and significantly improve the language processing accuracy of the language model.

[0047] The data processing method provided in the embodiment of the present application can be implemented by a computer device, which can be a terminal device or a server, wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, a smart TV, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0048] The data processing method provided in the embodiment of the present application is based on artificial intelligence. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0049] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0050] In the embodiments of the present application, the artificial intelligence technologies mainly involved include the above-mentioned natural language processing technologies and other directions.

[0051] The aforementioned computer processing device may have natural language processing capabilities. Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0052] In an embodiment of the present application, a computer device implements language processing on the corpus to be processed through a deployed language model. Based on the different functional types of the language model, different language processing functions can be implemented, such as machine translation, public opinion monitoring, automatic summarization, opinion extraction, text classification, question answering, text semantic comparison, speech recognition, Chinese OCR and other functions.

[0053] pass Figure 1 To explain, Figure 1 In the illustrated scenario, the aforementioned computer device takes the server 100 as an example, and the language model takes the translation model 200 (Transformer Model) for implementing a language translation function as an example.

[0054] The translation model 200 has an encoder and a decoder. The decoder is provided with n attention layers, which are attention layer 1 to attention layer n in the decoding order. For example, for ease of explanation, the first layer of the n attention layers, attention layer 1, is used as the first attention layer in this application, and the first attention mechanism is adopted for this layer. Attention layer 2 is used as the second attention layer in this application, and the second attention mechanism is adopted for this layer. There is no limitation on what kind of attention mechanism is adopted for the attention layers other than attention layer 1 and attention layer 2 in these n attention layers.

[0055] The translation model 200 obtained based on the above configuration is configured in the server 100. The server 100 provides users with translation services in different languages ​​based on the translation model 200. Moreover, the decoder of the translation model can use at least two attention mechanisms for decoding processing, with high accuracy.

[0056] When the server 100 receives the corpus to be processed, the translation function provided by the translation model performs language processing on the corpus to be processed to obtain a corresponding translation text. For example, when the translation model 200 is a Chinese-English translation model, the corpus to be processed may be "The weather is sonice today", and the corresponding translation text "The weather is sonice today" can be obtained through the translation model 200.

[0057] During the translation process, the attention layer 1 of the decoder of the translation model 200 obtains the first sub-input data determined according to the input data, and the attention layer 1 can decode the first sub-input data based on the first attention mechanism, and the obtained first sub-processing result reflects the first attention distribution of the first attention mechanism. Based on the first sub-processing result, the attention layer 2 continues to perform decoding processing based on the second attention mechanism, and the obtained second sub-processing result reflects the second attention distribution of the second attention mechanism under the first attention distribution.

[0058] Therefore, the language model can adopt at least two different attention mechanisms when processing the corpus to be processed. Since different attention mechanisms have different attention distributions or focuses on the same corpus, the language model can pay attention to more data that needs attention by integrating different attention distributions, thereby increasing processing diversity and significantly improving the language processing accuracy of the language model.

[0059] Figure 2 This is a method flow chart of a data processing method provided in an embodiment of the present application. In this embodiment, a server is mainly used as the aforementioned computer device for illustration. The method includes:

[0060] S201: Determine the multiple attention layers included in the decoder of the language model.

[0061] This application does not limit the language processing functions that can be realized by the language model involved. For example, the language model can realize any one of the functions of translation between different languages, text generation, language understanding, information extraction, and semantic representation. Based on the different language processing functions that can be realized by the language model, the corresponding language processing results can also change accordingly.

[0062] The multiple attention layers in the decoder are connected sequentially in the decoding order. The first attention layer and the second attention layer are two adjacent layers among the multiple attention layers. In the decoding order, the first attention layer is the previous layer of the second attention layer, that is, the output obtained by the first attention layer will be used as the input of the second attention layer, and the second attention layer will determine the output of this layer based on the input.

[0063] Optionally, the number of attention layers can be 6.

[0064] S202: Setting at least two attention mechanisms in the multi-layer attention layer.

[0065] Compared with the method in the related art where only one attention mechanism is set for the language model, the embodiment of the present application sets at least two attention mechanisms in the decoder of the same language model, such as a first attention mechanism and a second attention mechanism. It should be noted that the present application does not limit the number of attention mechanisms set.

[0066] A language model that uses an attention mechanism has a relatively simple attention distribution when performing language processing and is better at using the attention mechanism.

[0067] See also Figure 3 ,exist Figure 3 The attention mechanism used in the scenario shown is a multi-head self-attention mechanism, and the language model is specifically a translation model. The translation model includes an encoder and a decoder, where the encoder and the decoder each have a complex structure of N layers (usually 6 layers). The structure of each layer is consistent. Each layer of the encoder includes a multi-head self-attention module (Multi-Head Self-Attention Module) and a feed-forward fully connected (FFN, Feed Forward Network) module, and the output of each module needs to be processed by addition normalization (Add&Norm).

[0068] Each of the N attention layers of the decoder usually includes a mask multi-head self-attention module, a cross encoder and decoder multi-head self-attention module, also called a cross self-attention module, and a feed-forward fully connected module. Figure 3 In the scenario shown, each layer of the decoder is called a multi-head self-attention layer, which is similar to Figure 1 An attention layer in .

[0069] Through the sequential decoding process of N layers of multi-head self-attention layers, after being processed by linearization and softmax function, the prediction result for the text to be translated is obtained, that is, the translated text obtained based on the model.

[0070] Due to the difference in attention mechanisms, different attention mechanisms will determine different attention distributions for the same corpus to be processed, that is, they will guide the model to focus on different important features. Therefore, when at least two attention mechanisms are set in the decoder of a language model, during the decoding process, the decoder will pay attention to the important features identified by different attention mechanisms under the guidance of different attention mechanisms. This approach not only brings diversity, but also improves the quality of the language processing results finally obtained by the language model.

[0071] The following is an example of a language model used for translation between different languages. The attention distribution determined by different attention mechanisms will have overlapping parts and non-overlapping parts. The overlapping parts can reflect the corpus features that are less likely to be mistranslated (easy parts), and the non-overlapping parts can reflect the corpus features that different attention mechanisms consider to be easy to be mistranslated under their own attention mechanisms, or the uncommon corpus features that require special attention (difficult parts).

[0072] The translation model can thus identify the corpus features that are more important to translation, determined by different attention mechanisms based on the attention distribution, by using the decoding results of the attention layers of different attention mechanisms. This allows the translation model to have higher translation accuracy for corpus features that are easy to translate, and to pay more extensive and comprehensive attention to corpus features that are prone to translation errors, thereby reducing the probability of translation errors in error-prone corpus and improving translation quality overall.

[0073] The same is true for other types of language models, which will also have the effect of improving the quality of language processing results.

[0074] When setting the attention mechanism, different attention mechanisms can be set for the first attention layer and the second attention layer as adjacent attention layers, for example, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism. It should be emphasized that for the attention mixing scheme of multiple attention layers in the decoder provided in the embodiment of the present application, the present application does not limit the attention mechanisms adopted by all adjacent attention layers to be different, as long as at least one pair of adjacent attention layers adopts different attention mechanisms. For example, the attention mixing scheme may include the case where some adjacent attention layers adopt the same attention mechanism.

[0075] S203: Determine input data of the language model according to the corpus to be processed.

[0076] S204: Based on the input data, determine the language processing result of the corpus to be processed by using the language model.

[0077] During the process of the language model performing the language processing function corresponding to the model on the input data determined based on the corpus to be processed, since at least two attention mechanisms are set in the decoder of the language model, when the server processes the corpus through the language model, different attention mechanisms have different attention distributions or focuses on the same corpus. By integrating different attention distributions, the language model can pay attention to more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model.

[0078] Next, the translation scenario is used as an example for explanation. In a possible implementation, the language model is a language translation model, and S203 includes:

[0079] Performing text segmentation processing on the to-be-processed corpus in the first language, and determining the input data according to the obtained multiple corpus segments;

[0080] Accordingly, S204 includes:

[0081] Based on the input data, a translation text of the to-be-processed corpus in the second language is determined through the at least two attention mechanisms adopted by the language model.

[0082] The overall translation process is to first determine the text information of the corpus to be processed, divide the text information into individual sentences, and then perform word segmentation on each sentence. For example, "I love China" is segmented into "I / love / China". The segmented words are then sent as input data to the translation model, which outputs the translated text in other languages. After that, the punctuation marks, spaces, etc. of the translated text can be simply post-processed and returned to the user.

[0083] Still taking the translation scenario as an example, the application scenario of the embodiment of the present application is described. Figure 4 As shown, the user can trigger a translation request for the corpus to be processed through the terminal device, for example, by long pressing the corpus to be processed in the conversation interface of the social software, or by determining the corpus to be processed through a mini program, such as in Figure 5 In the illustrated scenario, user a long presses "The weather is great today" through the conversation interface of the social software, and generates a translation request for "The weather is great today", hoping to translate it into English.

[0084] After the language detection and translation request distribution module of the server obtains the translation request, it calls the translation model provided in the embodiment of the present application to translate the corpus to be processed, and obtains the translation result returned by the translation model. For example, if the corpus to be processed is "The weather is so nice today", the returned translation result is "The weather is so nice today". The language detection and translation request distribution module returns the translation result to the user's terminal device, and the terminal device provides the user with a translation service by displaying the translation result, such as Figure 5 As shown, “The weather is so nice today” is displayed below the conversation interface where “The weather is so nice today” is located.

[0085] In order to clearly illustrate the technical effects of different attention distributions on language processing, especially the advantages of a language model under a single attention mechanism, the roles of the first attention layer and the second attention layer in S204 are described in the process of obtaining the language processing result:

[0086] S2041: Determine the first sub-input data of the first attention layer based on the input data.

[0087] After the input data is input into the language model, it can reach the decoder after being encoded by the encoder. If the first attention layer is the first layer of multiple attention layers of the decoder, the first sub-input data of the first attention layer can be determined according to the output of the encoder. If the first attention layer is not the first layer of multiple attention layers of the decoder, the first sub-input data of the first attention layer can be determined according to the output of the encoder and the output of the previous attention layer of the first attention layer.

[0088] S2042: Perform decoding processing on the first sub-input data based on the first attention mechanism through the first attention layer, and use the obtained first sub-processing result as the second sub-input data of the second attention layer.

[0089] Since the first attention layer adopts the first attention mechanism among at least two attention mechanisms, the first sub-processing result obtained after the decoding processing of the first attention layer reflects the first attention distribution of the first attention mechanism. The first attention distribution is equivalent to instructing the subsequent attention layers which parts of the input data should be paid attention to when performing decoding processing.

[0090] S2043: Perform decoding processing based on the second attention mechanism on the second sub-input data through the second attention layer to obtain a second sub-processing result.

[0091] Since the second attention layer uses the second attention mechanism, which is a different type of attention mechanism from the first attention mechanism, when the attention mechanism is used for decoding, a second attention distribution based on the second attention mechanism can be made under the attention distribution identified by the first attention mechanism. Thus, at least two attention distributions are combined in the second sub-processing result, and the language model will refer to these two attention distributions before making the final language processing result, so that the referenced attention distributions are diverse.

[0092] Under the guidance of multiple attention distributions, the language model can process corpus features that are easy to process with higher accuracy, and pay more extensive and comprehensive attention to corpus features that are prone to processing errors, thereby reducing the probability of processing errors in error-prone corpus and improving the overall quality of language processing.

[0093] It can be seen that the decoder for determining the language model includes multiple attention layers, including the first attention layer and the second attention layer that are adjacent to each other in the decoding order. At least two attention mechanisms are set in the multiple attention layers, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism, so as to construct a language model that can be decoded using at least two attention mechanisms. For the corpus to be processed, the language model can be used to obtain the language processing result corresponding to the corpus to be processed based on the input data determined by the corpus to be processed. During the processing, the first attention layer of the decoder of the language model will obtain the first sub-input data determined according to the input data, and the first attention layer can decode it based on the first attention mechanism, and the first sub-processing result obtained reflects the first attention distribution of the first attention mechanism. On the basis of the first sub-processing result, the second attention layer continues to perform decoding processing based on the second attention mechanism, and the second sub-processing result obtained reflects the second attention distribution of the second attention mechanism under the first attention distribution. Therefore, when processing the corpus to be processed, the language model can adopt at least two different attention mechanisms. Different attention mechanisms have different attention distributions or focuses on the same corpus. By integrating different attention distributions, the language model can pay attention to more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model.

[0094] Next, this application is explained based on a specific attention mechanism.

[0095] In one possible implementation, the at least two attention mechanisms may include a multi-head self-attention mechanism and an average attention mechanism (AverageAttention).

[0096] The multi-head self-attention mechanism can be achieved by splitting a high-dimensional vector of dimension m into n vectors of dimension m / n, corresponding to n heads respectively. Then, self-attention calculation is performed on the vector under each head.

[0097] The average attention mechanism replaces the multi-head self-attention mechanism on the decoder side with a calculation mechanism that uses the average of historical state vectors as the attention output.

[0098] The computational complexity of the multi-head self-attention mechanism is the square of the length of the sentence (O(n 2 ), that is, as the length of the sentences in the corpus to be processed increases, the amount of computation of the multi-head self-attention mechanism will increase very quickly. In related technologies, this process is accelerated by quantizing the model and optimizing the details of the self-attention operator, but this cannot solve the limitation of the theoretical computational complexity of self-attention.

[0099] The model structure for implementing the multi-head attention mechanism can be found in Figure 6 , the multi-head self-attention mechanism divides an input vector into multiple small sub-vectors (such as V, K, and Q), each of which is sent to a different head to calculate the attention score, and finally the attention scores of each head are concatenated and summed. Figure 6 The number of heads is h=3, and each head is represented by scaled dot-product attention.

[0100] The calculation formula is as follows:

[0101]

[0102] MultiHead(Q,K,V)=Concat(head1,..,head h )W O

[0103]

[0104] Among them, d k is the dimension of the input vector, W O is the output matrix, W i is the output vector of the i-th head.

[0105] The theoretical computational complexity of the average attention mechanism and the multi-head self-attention mechanism is the same, both are O(n 2), but in actual translation, the amount of repeated calculations can be reduced by saving the past attention results. Each calculation can use the attention results of the previous time step as the calculation input, which can effectively reduce the theoretical amount of calculations in the actual decoding process to speed up the decoding. However, in most scenarios, the quality of language processing results of the average attention mechanism is slightly worse than that of the multi-head self-attention mechanism.

[0106] The model structure for implementing the multi-head attention mechanism can be found in Figure 7 , which includes an input layer, an average layer, and a gating layer. Each layer includes four processing nodes.

[0107] The calculation formula is as follows:

[0108]

[0109] i j ,f j =σ(W[y j ; g j ])

[0110]

[0111] Where j∈4, y j is the output of the input layer, g j is the output of the averaging layer, h j is the output of the gating layer, is the input of the gating layer, i j and f j are the weights of the input layer and the averaging layer.

[0112] That is to say, for the language model that adopts a single attention mechanism in the related technology, whether it is the aforementioned multi-head self-attention mechanism or the average attention mechanism, it is difficult to achieve a balance between time and quality in model quality optimization. Either the processing speed of the language model slows down, affecting the quality of online services, or the processing speed is fast, but the model quality is difficult to meet expectations.

[0113] In one possible implementation, the first attention mechanism is a multi-head self-attention mechanism, and the second attention mechanism is an average attention mechanism; or, the first attention mechanism is an average attention mechanism, and the second attention mechanism is a multi-head self-attention mechanism.

[0114] This application utilizes the characteristics of these two attention mechanisms. By mixing these two attention mechanisms, it is possible to complement each other and effectively achieve a balance between processing efficiency and processing quality. When mixing, the mixing effect can be controlled by controlling the number ratio of the attention layers that use the multi-head self-attention mechanism and the average attention mechanism, or by a specific mixing method, so that the processing efficiency and processing results of the constructed language model can meet expectations.

[0115] Regarding the mixing ratio, in a possible implementation, the ratio of the number of attention layers using the first attention mechanism to the number of attention layers using the second attention mechanism in the multi-layer attention layer is one to one. For example, if the decoder of the language model includes 6 attention layers, 3 of the attention layers can use the multi-head self-attention mechanism, and the other 3 attention layers can use the average attention mechanism.

[0116] Regarding the mixing method, two specific mixing methods are mainly introduced below.

[0117] The first hybrid method prioritizes the multi-head self-attention mechanism:

[0118] In a possible implementation, the first attention layer is the first attention layer in the multi-layer attention layer in the decoding order, and the first attention layer is a multi-head self-attention layer using a multi-head self-attention mechanism;

[0119] In the multi-layer attention layer, the next n attention layers after the first attention layer in the decoding order are average attention layers that adopt the average attention mechanism.

[0120] In this hybrid approach, a relatively high-accuracy multi-head self-attention mechanism is used in the first attention layer to obtain high-quality output, laying the foundation for the processing accuracy of subsequent attention layers. A model structure of this hybrid approach can be found in Figure 8 Optionally, n can be set to 3, with n+1 being less than or equal to the number of attention layers in the decoder.

[0121] In the translation scenario, the results are outstanding in the three major corpora of English-Chinese, English-German, and English-Japanese. Compared with the translation model using a single multi-head self-attention mechanism, it can stably improve 0.3 to 0.4 BLEU.

[0122] The second hybrid method, dual attention mechanism:

[0123] The embodiment of the present application also provides a method for using two attention mechanisms to perform calculations in one attention layer at the same time. In one possible implementation, the first attention layer is a hybrid attention layer that uses the first attention mechanism and the second attention mechanism.

[0124] The final attention score is obtained by taking the weighted sum of the attention scores calculated by the two attention mechanisms (multi-head self-attention mechanism and average attention mechanism).

[0125] Because the two have different attention calculation methods, the distribution of attention scores is different. This diversity allows the two to complement each other, making up for the problem of deviation in the calculation of a certain attention mechanism, and further improving the effect of the language model.

[0126] In one possible implementation, the first attention layer includes multiple component modules, wherein the first module in the decoding order includes a first submodule corresponding to the first attention mechanism and a second submodule corresponding to the second attention mechanism, and the first submodule and the second submodule are parallel structures.

[0127] In other words, the calculations of the two attention modules can be performed in parallel. When there are sufficient computing resources, the calculation time depends on the slower attention module, so the overall calculation speed is not significantly reduced. At the same time, it is also different from other models that only use one attention calculation per layer. It has good diversity and can be used for model integration.

[0128] It should be noted that, in the decoder of the language model deployed by the server, at least one layer of the multi-layer attention layer can be deployed in the second hybrid mode. Optionally, the second hybrid mode can be mechanically deployed in all multi-layer attention layers.

[0129] A model structure of this hybrid approach can be found in Fig. 9 ,It can be seen that in one attention layer, the multi-head self-attention mechanism module and the average attention module are set in parallel.

[0130] In addition to the two main hybrid methods introduced above, the present application embodiment also provides other hybrid methods, such as the average attention mechanism priority method. A model structure of this hybrid method can be seen in Fig.10 For example, the multi-head attention mechanism and the average attention mechanism are alternately deployed in the attention layer.

[0131] The attention-mixed model construction method provided in the embodiment of the present application does not increase the hardware cost of the constructed language model, and the hardware environment support during the use process is relatively simple, and training and deployment can be carried out online by providing an ordinary server environment. For details, please refer to Table 1.

[0132] Table 1 Brief description of software and hardware environment

[0133]

[0134]

[0135] When an average attention mechanism is included in at least two attention mechanisms, in order to improve the contribution of the average attention mechanism in the language model, an embodiment of the present application provides a method for adjusting the calculation weights in the average attention mechanism, thereby improving the accuracy of the average attention mechanism.

[0136] If the first attention mechanism is an average attention mechanism, for S2041, in a possible implementation, the method includes:

[0137] When decoding the jth word segment in the corpus to be processed, the sub-processing results of the first attention layer on the first j-1 word segments in the corpus to be processed are obtained.

[0138] The sub-processing result for the j-th participle is determined according to the sub-processing results of the first j-1 participles and the influence weights respectively corresponding to the first j-1 participles.

[0139] The size of the influence weight is inversely correlated with the distance between the corresponding participle and the jth participle in the corpus to be processed.

[0140] The specific calculation formula is as follows:

[0141]

[0142] g j =FFN(c j )

[0143] Among them, α is the coefficient and y0 is the first identifier of the input data.

[0144] Therefore, each time the attention result of the previous time step is used as the calculation input, for the jth time step, the influence of the attention result of the previous j-1 words on the attention calculation of the current word will be inversely correlated with the word distance, that is, the farther a word in the previous j-1 words is from the current word, the smaller the influence on the attention calculation of the current word. This makes the attention calculation of the current word more reasonable, and the amount of reference information can better conform to the law of semantic expression.

[0145] Fig.11A device structure diagram of a data processing device provided in an embodiment of the present application, wherein the data processing device 1100 includes a first determining unit 1101, a setting unit 1102, a second determining unit 1103, and a third determining unit 1104:

[0146] The first determining unit 1101 is used to determine the multiple attention layers included in the decoder of the language model, the multiple attention layers connected in sequence include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order;

[0147] The setting unit 1102 is used to set at least two attention mechanisms in the multi-layer attention layer, the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism;

[0148] The second determining unit 1103 is used to determine the input data of the language model according to the corpus to be processed;

[0149] The third determining unit 1104 is used to determine the language processing result of the corpus to be processed by the language model based on the input data;

[0150] In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the third determining unit 1104 is further used to:

[0151] Determine first sub-input data of the first attention layer based on the input data;

[0152] Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism;

[0153] The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

[0154] In a possible implementation, the first attention mechanism is a multi-head self-attention mechanism, and the second attention mechanism is an average attention mechanism; or,

[0155] The first attention mechanism is an average attention mechanism, and the second attention mechanism is a multi-head self-attention mechanism.

[0156] In a possible implementation, the first attention layer is the first attention layer in the multi-layer attention layer in the decoding order, and the first attention layer is a multi-head self-attention layer using a multi-head self-attention mechanism;

[0157] In the multi-layer attention layer, the next n attention layers after the first attention layer in the decoding order are average attention layers that adopt the average attention mechanism.

[0158] In one possible implementation, the first attention layer is a hybrid attention layer that uses the first attention mechanism and the second attention mechanism.

[0159] In one possible implementation, the first attention layer includes multiple component modules, wherein the first module in the decoding order includes a first submodule corresponding to the first attention mechanism and a second submodule corresponding to the second attention mechanism, and the first submodule and the second submodule are parallel structures.

[0160] In one possible implementation, the ratio of the number of attention layers using the first attention mechanism to the number of attention layers using the second attention mechanism in the multi-layer attention layer is one to one.

[0161] In a possible implementation manner, the language model is a language translation model, and the second determining unit is further configured to:

[0162] Performing text segmentation processing on the to-be-processed corpus in the first language, and determining the input data according to the obtained multiple corpus segments;

[0163] The determining, based on the input data, a language processing result of the corpus to be processed by the language model includes:

[0164] Based on the input data, a translation text of the to-be-processed corpus in the second language is determined through the at least two attention mechanisms adopted by the language model.

[0165] In a possible implementation manner, if the first attention mechanism is an average attention mechanism, the third determining unit is further used to:

[0166] When decoding the jth word segment in the corpus to be processed, obtaining the sub-processing results of the first attention layer on the first j-1 words segment in the corpus to be processed;

[0167] According to the sub-processing results of the first j-1 participles and the influence weights corresponding to the first j-1 participles, the sub-processing result for the j-th participle is determined, and the size of the influence weight is inversely correlated with the distance between the corresponding participle and the j-th participle in the corpus to be processed.

[0168] It can be seen that the decoder for determining the language model includes multiple layers of attention layers, including the first attention layer and the second attention layer that are adjacent to each other in the decoding order. At least two attention mechanisms are set in the multiple layers of attention layers, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism, so as to construct a language model that can be decoded using at least two attention mechanisms. For the corpus to be processed, the language model can be used to obtain the language processing result corresponding to the corpus to be processed based on the input data determined by the corpus to be processed. During the processing, the first attention layer of the decoder of the language model will obtain the first sub-input data determined according to the input data, and the first attention layer can decode it based on the first attention mechanism, and the first sub-processing result obtained reflects the first attention distribution of the first attention mechanism. On the basis of the first sub-processing result, the second attention layer continues to perform decoding processing based on the second attention mechanism, and the second sub-processing result obtained reflects the second attention distribution of the second attention mechanism under the first attention distribution. Therefore, when processing the corpus to be processed, the language model can adopt at least two different attention mechanisms. Different attention mechanisms have different attention distributions or focuses on the same corpus. By integrating different attention distributions, the language model can pay attention to more data that needs attention, increase processing diversity, and significantly improve the language processing accuracy of the language model.

[0169] The embodiment of the present application further provides a computer device, which is the computer device described above and may include a terminal device or a server, and the data processing device described above may be configured in the computer device. The computer device is described below in conjunction with the accompanying drawings.

[0170] If the computer device is a terminal device, see Fig.12 As shown, the embodiment of the present application provides a terminal device, taking the terminal device as a mobile phone as an example:

[0171] Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 1410, a memory 1420, an input unit 1430, a display unit 1440, a sensor 1450, an audio circuit 1460, a wireless fidelity (WiFi) module 1470, a processor 1480, and a power supply 1490. Those skilled in the art can understand that Fig.12The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0172] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:

[0173] The RF circuit 1410 can be used for receiving and sending signals during the process of sending and receiving information or making calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1480 for processing; in addition, the designed uplink data is sent to the base station. Usually, the RF circuit 1410 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, referred to as LNA), a duplexer, etc. In addition, the RF circuit 1410 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0174] The memory 1420 can be used to store software programs and modules. The processor 1480 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1420. The memory 1420 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1420 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0175] The input unit 1430 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1430 may include a touch panel 1431 and other input devices 1432. The touch panel 1431, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1431 or near the touch panel 1431 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1431 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1480, and can receive and execute commands sent by the processor 1480. In addition, the touch panel 1431 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1431, the input unit 1430 may also include other input devices 1432. Specifically, other input devices 1432 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, etc.

[0176] The display unit 1440 may be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1440 may include a display panel 1441. Optionally, the display panel 1441 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1431 may cover the display panel 1441. When the touch panel 1431 detects a touch operation on or near it, it is transmitted to the processor 1480 to determine the type of touch event. Subsequently, the processor 1480 provides a corresponding visual output on the display panel 1441 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 1431 and the display panel 1441 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1431 and the display panel 1441 can be integrated to realize the input and output functions of the mobile phone.

[0177] The mobile phone may also include at least one sensor 1450, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1441 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1441 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0178] The audio circuit 1460, the speaker 1461, and the microphone 1462 can provide an audio interface between the user and the mobile phone. The audio circuit 1460 can transmit the received audio data to the speaker 1461 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1462 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1460 and converted into audio data, and then the audio data is output to the processor 1480 for processing, and then sent to another mobile phone through the RF circuit 1410, or the audio data is output to the memory 1420 for further processing.

[0179] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1470. It provides users with wireless broadband Internet access. Fig.12 A WiFi module 1470 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0180] The processor 1480 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1420, and calling data stored in the memory 1420. Optionally, the processor 1480 may include one or more processing units; preferably, the processor 1480 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1480.

[0181] The mobile phone also includes a power supply 1490 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1480 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption management.

[0182] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0183] In this embodiment, the processor 1480 included in the terminal device also has the following functions:

[0184] Determine a decoder of the language model includes multiple attention layers, wherein the multiple attention layers connected sequentially include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order;

[0185] At least two attention mechanisms are set in the multi-layer attention layer, and the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the first attention layer adopts the first attention mechanism, and the second attention layer adopts the second attention mechanism;

[0186] Determining input data of the language model according to the corpus to be processed;

[0187] Based on the input data, determining a language processing result of the corpus to be processed by using the language model;

[0188] In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the process includes:

[0189] Determine first sub-input data of the first attention layer based on the input data;

[0190] Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism;

[0191] The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

[0192] If the computer device is a server, the present application embodiment also provides a server, see Fig.13 As shown, Fig.13The structural diagram of the server 1500 provided in the embodiment of the present application, the server 1500 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1522 (for example, one or more processors) and a memory 1532, and one or more storage media 1530 (for example, one or more mass storage devices) storing application programs 1542 or data 1544. Among them, the memory 1532 and the storage medium 1530 can be short-term storage or permanent storage. The program stored in the storage medium 1530 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 1522 can be configured to communicate with the storage medium 1530 and execute a series of instruction operations in the storage medium 1530 on the server 1500.

[0193] The server 1500 may also include one or more power supplies 1526, one or more wired or wireless network interfaces 1550, one or more input and output interfaces 1558, and / or one or more operating systems 1541, such as Windows Server 2003 or Windows Server 2008. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0194] The steps performed by the server in the above embodiment can be based on Fig.13 The server structure shown.

[0195] In addition, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, and the computer program is used to execute the method provided in the above embodiment.

[0196] The embodiments of the present application also provide a computer program product including instructions, which, when executed on a computer, enables the computer to execute the method provided in the above embodiments.

[0197] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium can be at least one of the following media: read-only memory (English: Read-only Memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc. Various media that can store program codes.

[0198] It should be noted that each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, in which the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0199] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, It is characterized in that The method comprises: Determine a decoder of the language model includes multiple attention layers, wherein the multiple attention layers connected sequentially include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order; At least two attention mechanisms are set in the multi-layer attention layer, the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the second attention layer adopts the second attention mechanism, the first attention layer is a mixed attention layer adopting the first attention mechanism and the second attention mechanism, the first attention layer includes a plurality of component modules, wherein the first module in the decoding order includes a first submodule corresponding to the first attention mechanism and a second submodule corresponding to the second attention mechanism, and the first submodule and the second submodule are parallel structures; Determining input data of the language model according to the corpus to be processed; Based on the input data, determining a language processing result of the corpus to be processed by using the language model; In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the process includes: Determine first sub-input data of the first attention layer based on the input data; Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism; The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

2. The method according to claim 1, It is characterized in that The method further comprises: The first attention mechanism is a multi-head self-attention mechanism, and the second attention mechanism is an average attention mechanism; or, The first attention mechanism is an average attention mechanism, and the second attention mechanism is a multi-head self-attention mechanism.

3. The method according to claim 2, It is characterized in that The first attention layer is the first attention layer in the multi-layer attention layer in the decoding order, and the first attention layer is a multi-head self-attention layer using a multi-head self-attention mechanism; In the multi-layer attention layer, the next n attention layers after the first attention layer in the decoding order are average attention layers that adopt the average attention mechanism.

4. The method according to any one of claims 1 to 3, It is characterized in that The ratio of the number of attention layers using the first attention mechanism to the number of attention layers using the second attention mechanism in the multi-layer attention layer is one to one.

5. The method according to any one of claims 1 to 3, It is characterized in that The language model is a language translation model, and the step of determining input data of the language model according to the corpus to be processed includes: Performing text segmentation processing on the to-be-processed corpus in the first language, and determining the input data according to the obtained multiple corpus segments; The determining, based on the input data, a language processing result of the corpus to be processed by the language model includes: Based on the input data, a translation text of the to-be-processed corpus in the second language is determined through the at least two attention mechanisms adopted by the language model.

6. The method according to any one of claims 1 to 3, It is characterized in that If the first attention mechanism is an average attention mechanism, performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, includes: When decoding the jth word segment in the corpus to be processed, obtaining the sub-processing results of the first attention layer on the first j-1 words segment in the corpus to be processed; According to the sub-processing results of the first j-1 participles and the influence weights corresponding to the first j-1 participles, the sub-processing result for the j-th participle is determined, and the size of the influence weight is inversely correlated with the distance between the corresponding participle and the j-th participle in the corpus to be processed.

7. A data processing device, It is characterized in that The device comprises a first determining unit, a setting unit, a second determining unit and a third determining unit: The first determining unit is used to determine the multiple attention layers included in the decoder of the language model, the multiple attention layers connected in sequence include a first attention layer and a second attention layer, and the first attention layer is a previous layer of the second attention layer in the decoding order; The setting unit is used to set at least two attention mechanisms in the multi-layer attention layer, the at least two attention mechanisms include a first attention mechanism and a second attention mechanism, the second attention layer adopts the second attention mechanism, the first attention layer is a mixed attention layer adopting the first attention mechanism and the second attention mechanism, the first attention layer includes a plurality of component modules, wherein the first module in the decoding order includes a first submodule corresponding to the first attention mechanism and a second submodule corresponding to the second attention mechanism, and the first submodule and the second submodule are parallel structures; The second determining unit is used to determine the input data of the language model according to the corpus to be processed; The third determining unit is used to determine the language processing result of the corpus to be processed through the language model based on the input data; In the process of obtaining the language processing result, for the first attention layer and the second attention layer, the third determining unit is further used to: Determine first sub-input data of the first attention layer based on the input data; Performing decoding processing based on the first attention mechanism on the first sub-input data through the first attention layer, and using the obtained first sub-processing result as the second sub-input data of the second attention layer, wherein the first sub-processing result reflects the first attention distribution of the first attention mechanism; The second sub-input data is decoded based on the second attention mechanism through the second attention layer to obtain a second sub-processing result, which reflects the second attention distribution of the second attention mechanism under the first attention distribution.

8. The device according to claim 7, It is characterized in that If the first attention mechanism is an average attention mechanism, the third determining unit is further used for: When decoding the jth word segment in the corpus to be processed, obtaining the sub-processing results of the first attention layer on the first j-1 words segment in the corpus to be processed; According to the sub-processing results of the first j-1 participles and the influence weights corresponding to the first j-1 participles, the sub-processing result for the j-th participle is determined, and the size of the influence weight is inversely correlated with the distance between the corresponding participle and the j-th participle in the corpus to be processed.

9. A computer device, It is characterized in that The computer device comprises a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the data processing method according to any one of claims 1 to 6 according to the instructions in the program code.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the data processing method according to any one of claims 1 to 6.

11. A computer program product, It is characterized in that The computer program product comprises instructions, and when the instructions are executed on a computer, the computer is caused to execute the data processing method according to any one of claims 1 to 6.

Images