Model training and application method, device, computer equipment and storage medium

Abstract

The present application relates to a model training method, apparatus, computer equipment, storage medium and computer program product. The method comprises: obtaining training sample data; constructing corresponding forward attention weight matrices and reverse attention weight matrices according to the training sample data, the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data; constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix; and training a preset model according to the bidirectional attention weight matrix. The present application can realize simultaneous bidirectional encoding, improve the training efficiency of the preset model, save hardware resources, increase hardware processing speed, and improve the accuracy of model-generated data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training and application method, apparatus, computer equipment, storage medium, and computer program product. Background Art

[0002] With the advancement of the times and the development of information technology, the internet has become an increasingly important platform for social interaction, entertainment, and even work in human life. It is the primary channel for people to access various knowledge resources. The internet is becoming an increasingly indispensable part of people's lives, penetrating into every aspect of life.

[0003] However, while the internet provides convenient and fast services, it also inevitably brings with it the problem of information overload. With the rapid growth of information data, the forms of information have also diversified, primarily including text, sound, and images. As the most basic form of information on the internet, text data, with its advantages of low resource consumption and easy transmission, has always occupied a key position in internet information dissemination. The ever-increasing volume of text information in various fields and forms has resulted in people searching for literature or materials, even after using search engines, often only finding an overwhelming amount of redundant information, requiring considerable time and effort to sift through. This creates a significant obstacle to knowledge acquisition. Presenting this vast and redundant text data to users in a concise and clear manner, thereby helping them quickly and efficiently obtain the information they need, has become a pressing issue.

[0004] Training the existing model with specific sample data to obtain a model that can generate target data is an important way to solve this problem. However, the accuracy of the target data generated by the trained model is currently low. Summary of the Invention

[0005] Based on this, it is necessary to provide a model training and application method, device, computer equipment, computer-readable storage medium and computer program product that can improve the accuracy of target data generated by the model to address the above technical problems.

[0006] In a first aspect, the present application provides a model training method. The method comprises:

[0007] Obtain training sample data;

[0008] According to the training sample data, a corresponding forward attention weight matrix and a reverse attention weight matrix are respectively constructed, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data;

[0009] Constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix;

[0010] The preset model is trained according to the bidirectional attention weight matrix.

[0011] In one embodiment, constructing the corresponding forward attention weight matrix and reverse attention weight matrix according to the training sample data includes:

[0012] Converting the training sample data into a sample matrix;

[0013] Determining the number of rows and columns of a first initial attention matrix according to the number of rows of the sample matrix, and determining the number of rows and columns of a second initial attention matrix;

[0014] Initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part along their own diagonals;

[0015] Covering the second part of the first initial attention matrix with a first mask template to obtain the forward attention weight matrix;

[0016] A second mask template is used to cover the first part of the second initial attention matrix to obtain the inverse attention weight matrix.

[0017] In one embodiment, constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix includes:

[0018] The forward attention weight matrix and the reverse attention weight matrix are concatenated and normalized to obtain the bidirectional attention weight matrix.

[0019] In one embodiment, constructing the corresponding forward attention weight matrix and reverse attention weight matrix according to the training sample data includes:

[0020] Converting the training sample data into a sample matrix;

[0021] Determining the number of rows and columns of a first initial attention matrix and the number of rows and columns of a second initial attention matrix according to the number of rows of the sample matrix;

[0022] Initialize the weight values ​​of the first initial attention matrix to obtain the corresponding forward attention weight matrix, and initialize the weight values ​​of the second initial attention matrix to obtain the corresponding reverse attention weight matrix.

[0023] In one embodiment, constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix includes:

[0024] Splicing the forward attention weight matrix and the reverse attention weight matrix to obtain a spliced ​​matrix; the forward attention weight matrix and the reverse attention weight matrix are respectively divided into a first part and a second part along their own diagonals;

[0025] The second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain the bidirectional attention weight matrix.

[0026] In one embodiment, masking the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix to obtain the bidirectional attention weight matrix includes:

[0027] The first mask template in the splicing matrix is ​​used to cover the second part of the forward attention weight matrix, and the second mask template is used to cover the first part of the reverse attention weight matrix to obtain the bidirectional attention weight matrix.

[0028] In one embodiment, the training of a preset model according to the bidirectional attention weight matrix includes:

[0029] Processing the training sample data according to the forward attention weight matrix to obtain a first loss;

[0030] Processing the training sample data according to the inverse attention weight matrix to obtain a second loss;

[0031] Processing the training sample data according to the bidirectional attention weight matrix to obtain a third loss;

[0032] A total loss is determined based on the first loss, the second loss, and the third loss, and the preset model is trained based on the total loss.

[0033] In a second aspect, the present application also provides a model training device. The device comprises:

[0034] Sample acquisition module, used to obtain training sample data;

[0035] A first construction module is used to construct corresponding forward attention weight matrices and reverse attention weight matrices according to the training sample data, wherein the forward attention weight matrix is ​​used to predict the next adjacent element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the previous adjacent element of each element in the training sample data;

[0036] A second construction module is used to construct a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix;

[0037] A model training module is used to train a preset model based on the bidirectional attention weight matrix.

[0038] In one embodiment, the first building block is further configured to:

[0039] Converting the training sample data into a sample matrix;

[0040] Determining the number of rows and columns of a first initial attention matrix according to the number of rows of the sample matrix, and determining the number of rows and columns of a second initial attention matrix;

[0041] Initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part along their own diagonals;

[0042] Covering the second part of the first initial attention matrix with a first mask template to obtain the forward attention weight matrix;

[0043] A second mask template is used to cover the first part of the second initial attention matrix to obtain the inverse attention weight matrix.

[0044] In one embodiment, the second building block is further configured to:

[0045] The forward attention weight matrix and the reverse attention weight matrix are concatenated and normalized to obtain the bidirectional attention weight matrix.

[0046] In one embodiment, the first building block is further configured to:

[0047] Converting the training sample data into a sample matrix;

[0048] Determining the number of rows and columns of a first initial attention matrix and the number of rows and columns of a second initial attention matrix according to the number of rows of the sample matrix;

[0049] Initialize the weight values ​​of the first initial attention matrix to obtain the corresponding forward attention weight matrix, and initialize the weight values ​​of the second initial attention matrix to obtain the corresponding reverse attention weight matrix.

[0050] In one embodiment, the second building block is further configured to:

[0051] Splicing the forward attention weight matrix and the reverse attention weight matrix to obtain a spliced ​​matrix; the forward attention weight matrix and the reverse attention weight matrix are respectively divided into a first part and a second part along their own diagonals;

[0052] The second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain the bidirectional attention weight matrix.

[0053] In one embodiment, the second building block is further configured to:

[0054] The first mask template in the splicing matrix is ​​used to cover the second part of the forward attention weight matrix, and the second mask template is used to cover the first part of the reverse attention weight matrix to obtain the bidirectional attention weight matrix.

[0055] In one embodiment, the model training module is further used to:

[0056] Processing the training sample data according to the forward attention weight matrix to obtain a first loss;

[0057] Processing the training sample data according to the inverse attention weight matrix to obtain a second loss;

[0058] Processing the training sample data according to the bidirectional attention weight matrix to obtain a third loss;

[0059] A total loss is determined based on the first loss, the second loss, and the third loss, and the preset model is trained based on the total loss.

[0060] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0061] Obtain training sample data;

[0062] According to the training sample data, a corresponding forward attention weight matrix and a reverse attention weight matrix are respectively constructed, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data;

[0063] Constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix;

[0064] The preset model is trained according to the bidirectional attention weight matrix.

[0065] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0066] Obtain training sample data;

[0067] According to the training sample data, a corresponding forward attention weight matrix and a reverse attention weight matrix are respectively constructed, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data;

[0068] Constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix;

[0069] The preset model is trained according to the bidirectional attention weight matrix.

[0070] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0071] Obtain training sample data;

[0072] According to the training sample data, a corresponding forward attention weight matrix and a reverse attention weight matrix are respectively constructed, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data;

[0073] Constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix;

[0074] The preset model is trained according to the bidirectional attention weight matrix.

[0075] The above-mentioned model training method, apparatus, computer equipment, storage medium and computer program product obtain training sample data; construct corresponding forward attention weight matrices and reverse attention weight matrices according to the training sample data, the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data; construct a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix; train the preset model according to the bidirectional attention weight matrix. The present application constructs a bidirectional attention weight matrix through the constructed forward attention weight matrix and the reverse attention weight matrix, and trains the preset model according to the bidirectional attention weight matrix, which can realize simultaneous bidirectional encoding, improve the training efficiency of the preset model, save hardware resources, improve hardware processing speed, and improve the accuracy of model-generated data.

[0076] In a sixth aspect, the present application provides a model application method. The method comprises:

[0077] Obtain the data to be predicted;

[0078] The model trained by the model training method is used to predict the data to be predicted to obtain a prediction result.

[0079] In a seventh aspect, the present application further provides a model application device. The device comprises:

[0080] A data acquisition module is used to obtain data to be predicted;

[0081] The data prediction module is used to predict the data to be predicted using a model trained by a model training method to obtain a prediction result.

[0082] In an eighth aspect, the present application further provides a computer device. The computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0083] Obtain the data to be predicted;

[0084] The model trained by the model training method is used to predict the data to be predicted to obtain a prediction result.

[0085] In a ninth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0086] Obtain the data to be predicted;

[0087] The model trained by the model training method is used to predict the data to be predicted to obtain a prediction result.

[0088] In a tenth aspect, the present application further provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements the following steps:

[0089] Obtain the data to be predicted;

[0090] The model trained by the model training method is used to predict the data to be predicted to obtain a prediction result.

[0091] The above-mentioned model application method, device, computer equipment, storage medium and computer program product obtain the data to be predicted, use the model trained by the model training method to predict the data to be predicted, and obtain the prediction result, which can improve the accuracy of the prediction result. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] Figure 1 This is an application environment diagram of a model training method in one embodiment;

[0093] Figure 2A A schematic diagram of noise reduction training encoding in one embodiment;

[0094] Figure 2B A schematic diagram of autoregressive training encoding in one embodiment;

[0095] Figure 3 Schematic diagram of a flow chart of a model training method in one embodiment;

[0096] Figure 4A Schematic diagram of forward attention weight matrix encoding in one embodiment;

[0097] Figure 4B Schematic diagram of reverse attention weight matrix encoding in one embodiment;

[0098] Figure 5 Schematic diagram of encoding of a bidirectional attention weight matrix in one embodiment;

[0099] Figure 6 304 is a flowchart of an embodiment;

[0100] Figure 7A A schematic diagram of a first initial attention matrix divided into a first part and a second part along its own diagonal line in one embodiment;

[0101] Figure 7B Schematic diagram of a second initial attention matrix divided into a first part and a second part along its own diagonal line in one embodiment;

[0102] Figure 8A A schematic diagram of an embodiment in which the first mask template covers the second part of the first initial attention matrix to obtain a forward attention weight matrix;

[0103] Figure 8B A schematic diagram of an embodiment in which a second mask template covers a first portion of a second initial attention matrix to obtain a reverse attention weight matrix;

[0104] Figure 9 is a flow chart of step 304 in another embodiment;

[0105] Figure 10 306 is a flowchart of an embodiment;

[0106] Figure 11 is a schematic diagram of step 1002 in one embodiment;

[0107] Figure 12 A schematic diagram of constructing a bidirectional attention weight matrix in one embodiment;

[0108] Figure 13 308 is a flowchart of an embodiment;

[0109] Figure 14 is a flow chart of step 308 in another embodiment;

[0110] Figure 15 Schematic diagram of a flow chart of a model application method in one embodiment;

[0111] Figure 16 Schematic diagram of a flow chart of a model training and application method in one embodiment;

[0112] Figure 17 is a structural block diagram of a model training device in one embodiment;

[0113] Figure 18 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0114] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0115] The model training method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. The server 104 receives the text training sample data sent by the terminal 102, and constructs corresponding forward attention weight matrices and reverse attention weight matrices based on the text training sample data, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the text training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample text data; a bidirectional attention weight matrix is ​​constructed based on the forward attention weight matrix and the reverse attention weight matrix; the server 104 trains the preset model based on the bidirectional attention weight matrix. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented as an independent server or a server cluster consisting of multiple servers.

[0116] It should be noted that any step in the model training method disclosed in the embodiments of this application can be implemented by terminal 102 and server 104 in an interactive manner, or by server 104 alone, or by terminal 102 alone, without specific limitation. It is understood that text training sample data can be replaced with speech training sample data or image training sample data.

[0117] Currently, most natural language models used in pre-training adopt the Transformer structure, using noise reduction training or unidirectional autoregressive training. Among them, the noise reduction training method uses mask identification to allow the model to complete bidirectional encoding, for example, encoding natural language sentences from left to right and from right to left, such as the BERT (Bidirectional Encoder Representation from Transformers) pre-training model; the unidirectional autoregressive training method can only perform unidirectional encoding, such as encoding from left to right, such as the GPT (Generative Pre-Training) pre-training model. The specific noise reduction training method encoding diagram is as follows Figure 2A As shown, the encoding diagram of the autoregressive training method is as follows Figure 2B shown.

[0118] Since the encoding of autoregressive training is unidirectional, the encoding efficiency is low. In order to solve the disadvantage of low encoding efficiency of autoregressive training and to avoid the problem of information leakage caused by the multi-layer stacked Transformer structure when directly performing bidirectional encoding, a noise reduction training encoding method is proposed. Although the noise reduction training method can achieve bidirectional encoding, during training, the encoded text in the sentence will be replaced with a pre-agreed identifier, such as a mask identifier, which will cause new problems: (1) In a text sentence, only the words or phrases replaced with identifiers will be encoded. In order to retain the general meaning of the sentence, not all texts will be replaced. Usually, only about 15% of the words or phrases will be replaced, so the training efficiency is still relatively low; (2) Since the identifier symbol is introduced during training, when fine-tuning is performed after the model training is completed, the input text does not have this identifier symbol, which will lead to inconsistency in the text used in the training and fine-tuning stages, thereby reducing the effectiveness of the training model.

[0119] In one embodiment, Figure 3 As shown, a model training method is provided, which is applied to Figure 1 The following steps are used as an example to illustrate the server in the example:

[0120] Step 302: Obtain training sample data.

[0121] The server can receive training sample data sent by the terminal or directly obtain it from the server interface. The training sample data can be in the form of text, speech, images, or other data. For example, each acquired training sample data is data corresponding to a text line, a speech segment, or an image. After obtaining the training sample data, it is preprocessed and converted into a matrix format before subsequent processing. Preprocessing includes denoising the training sample data and unifying the data size.

[0122] In step 304, a corresponding forward attention weight matrix and a reverse attention weight matrix are constructed based on the training sample data, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data.

[0123] For the same training sample data, the corresponding forward attention weight matrix and reverse attention weight matrix are constructed respectively. Among them, the forward attention weight matrix is ​​used to predict the next element adjacent to each element in the training sample data. For example, for text training sample data, the forward attention weight matrix can be used to predict the next word adjacent to each word in the text training sample data. Figure 4AAs shown; for the speech training sample data, the forward attention weight matrix can be used to predict the spectral features of the adjacent next preset period in each preset period of the spectral features; for the image training sample data, the forward attention weight matrix can be used to predict the next adjacent frame of each frame of the image training sample data. Similarly, the reverse attention weight matrix is used to predict the adjacent previous element of each element in the training sample data. For example, for the text training sample data, the reverse attention weight matrix can be used to predict the previous adjacent character of each character in the text training sample data, such as Figure 4B As shown; for the speech training sample data, the reverse attention weight matrix can be used to predict the spectral features of the adjacent previous preset period in each preset period of the spectral features; for the image training sample data, the reverse attention weight matrix can be used to predict the previous adjacent frame of each frame of the image training sample data.

[0124] The idea of attention is to represent the features of the elements in the training sample data as a weighted sum of the features of all elements in the training sample data. For example, for the text training sample data, the idea of attention is to represent the features of the characters in the sentence as a weighted sum of the features of all characters in the sentence. Attention can be understood as which elements in the training sample data the current element "pays attention to". The greater the weight of the corresponding element, the more attention it represents. For example, in the text sample training data, attention can be understood as which characters in the sentence the current character "pays attention to". The greater the weight of the corresponding character, the more "attention" it represents. The sum of the weights of all characters used for prediction is 1. When a mask is added to the attention, that is Figure 4A and Figure 4B the white circles in represent that the weight value at this position is artificially set to 0. The purpose of this is to not include the features of the characters at the mask positions during the weighted calculation.

[0125] For example Figure 4A in the forward attention weight matrix shown, a "upper triangular" mask is added to the attention weight matrix, that is, the features of the characters after the current character are masked to prevent information leakage. It should be noted here that when the content to be predicted appears in the features used for prediction, information leakage occurs. For example, when predicting the character "今", the character "今" also appears in the prediction content, that is, using the character "今" to predict the character "今", information leakage will occur. In this example, using the character "今" to predict the next character "天", that is H 天 =1.0*H 今 +0*H 天 +0*H 天 +0*H 气 +0*H 不 +0*H 错 , and again, using the characters before the character "气" to predict the next character "不", that is H不 = 0.3 * H 今 + 0.2 * H 天 + 0.1 * H 天 + 0.4 * H 气 + 0 * H 不 + 0 * H 错 。 Figure 4B In the reverse attention weight matrix shown, a "lower triangular" mask is added to the attention weight matrix, that is, the features of the words before the current word are masked. For example, use the words after the word "今" to predict the word "天" after the word "今".

[0126] Step 306, construct a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix.

[0127] In this embodiment, the server constructs a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix. The constructed bidirectional attention weight matrix can achieve masking only the word at its own position, that is, preventing information leakage, and at the same time using the features of as many other words as possible for both forward prediction and negative prediction, that is, achieving bidirectional encoding.

[0128] In a possible implementation, the Figure 4A example forward attention weight matrix and Figure 4B the word at its own position in the example reverse attention weight matrix are covered with a mask and then spliced to obtain the bidirectional attention weight matrix. As Figure 5 shown, for example, use the other words except the word "今" to predict the word "今", that is:

[0129]

[0130] Use the other words except the word "气" to predict the word "气", that is:

[0131]

[0132] Step 308, train the preset model according to the bidirectional attention weight matrix.

[0133] The server trains the preset model according to the bidirectional attention weight matrix. Among them, the preset model can be a model based on the transformer structure, such as the BERT model.

[0134] The above-mentioned model training method obtains training sample data; constructs corresponding forward attention weight matrices and reverse attention weight matrices according to the training sample data, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data; constructs a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix; and trains the preset model according to the bidirectional attention weight matrix. The embodiment of the present application constructs a bidirectional attention weight matrix through the constructed forward attention weight matrix and the reverse attention weight matrix, and trains the preset model according to the bidirectional attention weight matrix, which can realize simultaneous bidirectional encoding, improve the training efficiency of the preset model, save hardware resources, improve hardware processing speed, and improve the accuracy of model-generated data.

[0135] In one embodiment, Figure 6 As shown, in step 304, the corresponding forward attention weight matrix and reverse attention weight matrix are constructed according to the training sample data, including:

[0136] Step 602: Convert the training sample data into a sample matrix.

[0137] In this embodiment, the server converts the training sample data into a sample matrix, which is equivalent to digitizing the training sample data and converting it into a matrix. For example, for text training sample data, each word in a sentence can be converted into a corresponding word vector, thereby obtaining a word embedding matrix corresponding to the entire sentence; for image training sample data, a matrix representing the corresponding frame image can be obtained based on the individual pixels that make up the image; for speech training sample data, the spectral features of the speech training sample data are extracted, and the spectral features of the speech training sample data can be processed as an image, and then the image is converted into a corresponding matrix.

[0138] Step 604: Determine the number of rows and columns of the first initial attention matrix according to the number of rows of the sample matrix, and determine the number of rows and columns of the second initial attention matrix.

[0139] In this embodiment, the server determines the number of rows and columns of the first initial attention matrix based on the number of rows of the sample matrix, and determines the number of rows and columns of the second initial attention matrix based on the number of rows of the sample matrix. The number of rows and columns of the first initial attention matrix is ​​the same as the number of rows and columns of the second initial attention matrix.

[0140] In an optional embodiment, the sample matrix is ​​multiplied by the transposed matrix of the sample matrix, and the resulting matrix size corresponds to the size of the first initial attention matrix and the second initial attention matrix. That is, the first initial attention matrix and the second initial attention matrix are both square matrices. The number of rows of the sample matrix can be used as the number of rows and columns of the first initial attention matrix. Similarly, the number of rows of the sample matrix can be used as the number of rows and columns of the second initial attention matrix.

[0141] Step 606, initialize the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; wherein the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part by their own diagonals.

[0142] In this embodiment, when the preset model is trained for the first time, the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix need to be initialized. When the model training is performed later, the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix are continuously optimized according to the loss function. When initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix, they can be initialized according to the expectation and variance of the weight values. For example, the expectation of the weight values ​​of the first initial attention matrix is ​​controlled to be 0, and the expectation of the weight values ​​of the second initial attention matrix is ​​controlled to be 0, and the variance of the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix are the same.

[0143] The first initial attention matrix and the second initial attention matrix are divided into the first part and the second part respectively by their own diagonals. The training sample data is the text "The weather is good today" as an example. Figure 7A As shown, the first initial attention matrix is ​​divided into a first part 704 and a second part 706 by its main diagonal 702; Figure 7B As shown, the second initial attention matrix is ​​divided into a first part 710 and a second part 712 by its own main diagonal 708.

[0144] Step 608: Use the first mask template to cover the second part of the first initial attention matrix to obtain a forward attention weight matrix.

[0145] When covering a continuous area, a mask template with a fixed shape and size set in advance is usually used for covering. In this embodiment, the first mask template is used to cover the second part of the first initial attention matrix to obtain the forward attention weight matrix. Figure 8AAs shown, the first mask template is an isosceles right triangle, and the hypotenuse of the isosceles right triangle is the main diagonal of the first initial attention matrix, that is, the weight values ​​on the main diagonal of the first initial attention matrix are also covered by the first mask template. The first template of the isosceles right triangle is used to cover the second part of the first initial attention matrix to obtain the forward attention weight matrix.

[0146] Step 610: Use the second mask template to cover the first part of the second initial attention matrix to obtain a reverse attention weight matrix.

[0147] In this embodiment, the second mask template is used to cover the first part of the second initial attention matrix to obtain the reverse attention weight matrix. Figure 8B As shown, the second mask template is an isosceles right triangle, and the hypotenuse of the isosceles right triangle is the main diagonal of the second initial attention matrix, that is, the weight values ​​on the main diagonal of the second initial attention matrix are also covered by the second mask. The second template of the isosceles right triangle is used to cover the first part of the second initial attention matrix to obtain the reverse attention weight matrix.

[0148] In one embodiment, step 306 of constructing a bidirectional attention weight matrix based on the forward attention weight matrix and the reverse attention weight matrix includes:

[0149] The forward attention weight matrix and the reverse attention weight matrix are concatenated and normalized to obtain the bidirectional attention weight matrix.

[0150] In this embodiment, the forward attention weight matrix and the reverse attention weight matrix are spliced ​​to obtain a spliced ​​matrix, and the weight values ​​in the spliced ​​matrix are normalized to obtain the following: Figure 5 The bidirectional attention weight matrix shown in Figure 1 is normalized to convert the weight values ​​in the concatenated matrix into values ​​between 0 and 1. For example, the softmax function can be used for normalization. The softmax function is shown in the following formula:

[0151]

[0152] Among them, P(Z i,j ) represents the result of normalization processing, Z i,j is the weight value in the concatenation matrix, and C is the total number of weight values.

[0153] In one embodiment, Figure 9 As shown, in step 304, the corresponding forward attention weight matrix and reverse attention weight matrix are constructed according to the training sample data, including:

[0154] Step 902: Convert the training sample data into a sample matrix.

[0155] Step 904: Determine the number of rows and columns of the first initial attention matrix and the number of rows and columns of the second initial attention matrix based on the number of rows of the sample matrix.

[0156] Step 906: Initialize the weight values ​​of the first initial attention matrix to obtain the corresponding forward attention weight matrix, and initialize the weight values ​​of the second initial attention matrix to obtain the corresponding reverse attention weight matrix.

[0157] In this embodiment, steps 902 to 904 correspond to steps 602 to 604, and the specific description can be referred to steps 602 to 604. In this embodiment, the training sample data is converted into a sample matrix, and the number of rows and columns of the first initial attention matrix and the number of rows and columns of the second initial attention matrix are determined according to the number of rows of the sample matrix, and the weight values ​​of the first initial attention matrix are initialized to obtain the corresponding forward attention weight matrix, and the weight values ​​of the second initial attention matrix are initialized to obtain the corresponding reverse attention weight matrix. The difference from the embodiment described in steps 602 to 610 is that in this embodiment, the weights of the first initial attention matrix are initialized as the forward attention weight matrix, and the weight values ​​of the second initial attention matrix are initialized as the reverse attention weight matrix, and no corresponding mask template is used to cover some of the weight values.

[0158] In one embodiment, Figure 10 As shown, step 306 of constructing a bidirectional attention weight matrix according to the forward attention weight matrix and the reverse attention weight matrix includes:

[0159] Step 1002: Concatenate the forward attention weight matrix and the reverse attention weight matrix to obtain a concatenated matrix; wherein the forward attention weight matrix and the reverse attention weight matrix are respectively divided into a first part and a second part by their own diagonals.

[0160] In a specific example, Figure 11 As shown, the forward attention weight matrix and the reverse attention weight matrix are spliced ​​left and right to obtain a spliced ​​matrix; the forward attention weight matrix is ​​divided into a first part 1102 and a second part 1104 along its own diagonal, and the reverse attention weight matrix is ​​divided into a first part 1106 and a second part 1108 along its own diagonal.

[0161] In step 1004, the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain a bidirectional attention weight matrix.

[0162] In this embodiment, the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain a bidirectional attention weight matrix. The mask shapes and sizes used for the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix can be the same or different.

[0163] In one embodiment, the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain the bidirectional attention weight matrix, including:

[0164] The first mask template is used to cover the second part of the forward attention weight matrix in the splicing matrix, and the second mask template is used to cover the first part of the reverse attention weight matrix to obtain a bidirectional attention weight matrix.

[0165] In one possible implementation, Figure 12 As shown, in the splicing matrix, the first mask template is used to cover the second part of the forward attention weight matrix (see the white dot position part in the figure); the second mask template is used to cover the first part of the reverse attention weight matrix (see the white dot position part in the figure) to obtain a bidirectional attention weight matrix without its own information.

[0166] In one embodiment, Figure 13 As shown, step 308 of training the preset model according to the bidirectional attention weight matrix includes:

[0167] Step 1302: Process the training sample data according to the forward attention weight matrix to obtain a first loss.

[0168] Step 1304: Process the training sample data according to the inverse attention weight matrix to obtain a second loss.

[0169] Step 1306: Process the training sample data according to the bidirectional attention weight matrix to obtain a third loss.

[0170] Step 1308: Determine the total loss based on the first loss, the second loss, and the third loss, and train the preset model based on the total loss.

[0171] In this embodiment, for the same training sample data, the training sample data is processed simultaneously according to the forward attention weight matrix to obtain a first loss; the training sample data is processed according to the reverse attention weight matrix to obtain a second loss; and the training sample data is processed according to the bidirectional attention weight matrix to obtain a third loss. Based on the first loss, the second loss, and the third loss, a total loss is determined, and then the preset model is trained based on the total loss. Specifically, the first loss, the second loss, and the third loss can be weighted and summed to obtain the total loss, and the preset model can be trained based on the total loss.

[0172] In one possible implementation, Figure 14 As shown, the training sample data "The weather is good today" is processed according to the forward attention weight matrix, the reverse attention weight matrix and the bidirectional attention weight matrix at the same time, and the forward encoding, reverse encoding and bidirectional encoding of the text are realized at the same time. The same processing is performed on each layer of the preset model, and the preset model is trained comprehensively according to the respective loss functions corresponding to the forward encoding, reverse encoding and bidirectional encoding. This can improve the training accuracy of the preset model, thereby improving the accuracy of the target data generated by the trained preset model.

[0173] In one embodiment, Figure 15 As shown, a model application method is provided, which is described by taking the application of the method to a server as an example. The method includes the following steps:

[0174] Step 1502: Obtain data to be predicted.

[0175] The server obtains the data to be predicted, wherein the data to be predicted can be any one of text data, voice data or image data.

[0176] Step 1504: Use the model trained by the model training method to predict the data to be predicted to obtain a prediction result.

[0177] The server uses a model trained using a model training method to predict the data to be predicted and obtain a prediction result. The data to be predicted is input into the model trained using the model training method for prediction to obtain a prediction result. The prediction result can be data with similar semantics or meaning to the data to be predicted, or data with opposite semantics or meaning to the data to be predicted. For example, if the data to be predicted is text data, the prediction result can be a number of text data with similar semantics or text data with opposite semantics; if the data to be predicted is voice data, the prediction result can be a number of voice data with similar semantics or voice data with opposite semantics; if the data to be predicted is image data, the prediction result can be a number of similar image data or opposite image data.

[0178] The above-mentioned model application method obtains the data to be predicted, and uses the model trained by the model training method to predict the data to be predicted to obtain the prediction result, thereby improving the accuracy of the prediction result.

[0179] In one embodiment, Figure 16 As shown, the model training and application method is described by applying the method to a server, and includes the following steps:

[0180] Step 1602: Obtain training sample data.

[0181] The server obtains training sample data, which may be text data, voice data, or image data.

[0182] Step 1604: Convert the training sample data into a sample matrix, determine the number of rows and columns of the first initial attention matrix according to the number of rows of the sample matrix, and determine the number of rows and columns of the second initial attention matrix.

[0183] The server digitizes the training sample data and converts it into a sample matrix. In this embodiment, the number of rows and columns of the first initial attention matrix is ​​the same as the number of rows and columns of the second initial attention matrix, and is equal to the number of rows of the sample matrix.

[0184] Step 1606, initialize the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; wherein the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part by their own diagonals.

[0185] When initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix, they can be initialized according to the expectation and variance of the weight values. For example, the expectation of the weight values ​​of the first initial attention matrix is ​​controlled to be 0, and the expectation of the weight values ​​of the second initial attention matrix is ​​controlled to be 0, and the variance of the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix are the same.

[0186] The first initial attention matrix and the second initial attention matrix are divided into the first part and the second part respectively by their own diagonals. The training sample data is the text "The weather is good today" as an example. Figure 7A As shown, the first initial attention matrix is ​​divided into a first part 704 and a second part 706 by its main diagonal 702; Figure 7B As shown, the second initial attention matrix is ​​divided into a first part 710 and a second part 712 by its own main diagonal 708.

[0187] Step 1608: Use the first mask template to cover the second part of the first initial attention matrix to obtain the forward attention weight matrix; use the second mask template to cover the first part of the second initial attention matrix to obtain the reverse attention weight matrix.

[0188] The steps in this embodiment can be found in the description of step 608 and step 610, which will not be repeated here.

[0189] Step 1610: Concatenate the forward attention weight matrix and the reverse attention weight matrix, and perform normalization to obtain a bidirectional attention weight matrix.

[0190] The forward attention weight matrix and the reverse attention weight matrix are spliced ​​together to obtain a spliced ​​matrix, and the weight values ​​in the spliced ​​matrix are normalized to obtain Figure 5 The bidirectional attention weight matrix shown.

[0191] Step 1612: Process the training sample data according to the forward attention weight matrix to obtain a first loss; process the training sample data according to the reverse attention weight matrix to obtain a second loss; process the training sample data according to the bidirectional attention weight matrix to obtain a third loss; determine the total loss based on the first loss, the second loss, and the third loss, and train the preset model based on the total loss.

[0192] In this embodiment, the first loss, the second loss, and the third loss may be weightedly summed to obtain the total loss, and the preset model may be trained based on the total loss.

[0193] Step 1614: Obtain the data to be predicted.

[0194] The data to be predicted may be any one of text data, voice data or image data. The data to be predicted may also be the training sample data in step 1402.

[0195] Step 1616: Use the preset model trained in step 1612 to predict the data to be predicted to obtain a prediction result.

[0196] The data to be predicted is input into the model trained in step 1612 for prediction to obtain a prediction result. The prediction result can be data with similar semantics or meaning to the data to be predicted, or data with opposite semantics or meaning to the data to be predicted.

[0197] The above-mentioned model training and application method constructs a bidirectional attention weight matrix through the constructed forward attention weight matrix and reverse attention weight matrix, and trains the preset model according to the bidirectional attention weight matrix, which can realize simultaneous bidirectional encoding, improve the training efficiency and training accuracy of the preset model, save hardware resources, improve hardware processing speed, and improve the accuracy of model-generated data. It should be understood that although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

[0198] Based on the same inventive concept, the present application also provides a model training device for implementing the aforementioned model training method. The solution to the problem provided by the device is similar to the solution described in the aforementioned method. Therefore, the specific limitations in one or more of the following model training device embodiments can be found in the above-mentioned limitations on the model training method, and will not be repeated here.

[0199] In one embodiment, Figure 17 As shown, a model training device is provided, including: a sample acquisition module 1702, a first construction module 1704, a second construction module 1706 and a model training module 1708, wherein:

[0200] The sample acquisition module 1702 is used to obtain training sample data;

[0201] A first construction module 1704 is configured to construct a corresponding forward attention weight matrix and a reverse attention weight matrix based on the training sample data, wherein the forward attention weight matrix is ​​used to predict the next adjacent element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the previous adjacent element of each element in the training sample data;

[0202] A second construction module 1706 is configured to construct a bidirectional attention weight matrix based on the forward attention weight matrix and the reverse attention weight matrix;

[0203] The model training module 1708 is used to train the preset model according to the bidirectional attention weight matrix.

[0204] In one embodiment, the first building module 1704 is further configured to:

[0205] Converting the training sample data into a sample matrix;

[0206] Determining the number of rows and columns of a first initial attention matrix according to the number of rows of the sample matrix, and determining the number of rows and columns of a second initial attention matrix;

[0207] Initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part along their own diagonals;

[0208] Covering the second part of the first initial attention matrix with a first mask template to obtain the forward attention weight matrix;

[0209] A second mask template is used to cover the first part of the second initial attention matrix to obtain the inverse attention weight matrix.

[0210] In one embodiment, the second building module 1706 is further configured to:

[0211] The forward attention weight matrix and the reverse attention weight matrix are concatenated and normalized to obtain the bidirectional attention weight matrix.

[0212] In one embodiment, the first building module 1704 is further configured to:

[0213] Converting the training sample data into a sample matrix;

[0214] Determining the number of rows and columns of a first initial attention matrix and the number of rows and columns of a second initial attention matrix according to the number of rows of the sample matrix;

[0215] Initialize the weight values ​​of the first initial attention matrix to obtain the corresponding forward attention weight matrix, and initialize the weight values ​​of the second initial attention matrix to obtain the corresponding reverse attention weight matrix.

[0216] In one embodiment, the second building module 1706 is further configured to:

[0217] Splicing the forward attention weight matrix and the reverse attention weight matrix to obtain a spliced ​​matrix; the forward attention weight matrix and the reverse attention weight matrix are respectively divided into a first part and a second part along their own diagonals;

[0218] The second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain the bidirectional attention weight matrix.

[0219] In one embodiment, the second building module 1706 is further configured to:

[0220] The first mask template in the splicing matrix is ​​used to cover the second part of the forward attention weight matrix, and the second mask template is used to cover the first part of the reverse attention weight matrix to obtain the bidirectional attention weight matrix.

[0221] In one embodiment, the model training module 1708 is further configured to:

[0222] Processing the training sample data according to the forward attention weight matrix to obtain a first loss;

[0223] Processing the training sample data according to the inverse attention weight matrix to obtain a second loss;

[0224] Processing the training sample data according to the bidirectional attention weight matrix to obtain a third loss;

[0225] A total loss is determined based on the first loss, the second loss, and the third loss, and the preset model is trained based on the total loss.

[0226] Based on the same inventive concept, the present application also provides a model application device for implementing the aforementioned model application method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more model application device embodiments provided below can be found in the above-mentioned limitations of the model application method and will not be repeated here.

[0227] In one embodiment, a model application device is provided, comprising:

[0228] A data acquisition module is used to obtain data to be predicted;

[0229] The data prediction module is used to predict the data to be predicted using a model trained by a model training method to obtain a prediction result.

[0230] Each module in the above-mentioned model training device or model application device can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0231] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 18 As shown. The computer device includes a processor, a memory and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store attention weight data. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a model training method or a model application method is implemented.

[0232] Those skilled in the art will understand that Figure 18 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0233] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps of the model training method in the above embodiment are implemented.

[0234] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model training method in the above embodiment are implemented.

[0235] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the model training method in the above embodiment.

[0236] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the model application method in the above embodiment when executing the computer program.

[0237] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the model application method in the above embodiment are implemented.

[0238] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the model application method in the above embodiment when executed by a processor.

[0239] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0240] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0241] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A model training method, characterized in that: The method comprises: Obtain training sample data; According to the training sample data, a corresponding forward attention weight matrix and a reverse attention weight matrix are respectively constructed, wherein the forward attention weight matrix is ​​used to predict the adjacent next element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the adjacent previous element of each element in the training sample data; Constructing a bidirectional attention weight matrix without self-information based on the forward attention weight matrix and the reverse attention weight matrix; The training sample data is processed according to the forward attention weight matrix to obtain a first loss; the training sample data is processed according to the reverse attention weight matrix to obtain a second loss; the training sample data is processed according to the bidirectional attention weight matrix to obtain a third loss; A total loss is determined based on the first loss, the second loss, and the third loss, and a preset model is trained based on the total loss.

2. The method according to claim 1, characterized in that The method of constructing a corresponding forward attention weight matrix and a reverse attention weight matrix according to the training sample data includes: Converting the training sample data into a sample matrix; Determining the number of rows and columns of a first initial attention matrix according to the number of rows of the sample matrix, and determining the number of rows and columns of a second initial attention matrix; Initializing the weight values ​​of the first initial attention matrix and the weight values ​​of the second initial attention matrix; the first initial attention matrix and the second initial attention matrix are respectively divided into a first part and a second part along their own diagonals; Using a first mask template to cover the second part of the first initial attention matrix to obtain a forward attention weight matrix; A second mask template is used to cover the first part of the second initial attention matrix to obtain a reverse attention weight matrix.

3. The method according to claim 2, characterized in that The constructing of a bidirectional attention weight matrix without self-information based on the forward attention weight matrix and the reverse attention weight matrix includes: The forward attention weight matrix and the reverse attention weight matrix are concatenated and normalized to obtain a bidirectional attention weight matrix without its own information.

4. The method according to claim 1, wherein The method of constructing a corresponding forward attention weight matrix and a reverse attention weight matrix according to the training sample data includes: Converting the training sample data into a sample matrix; Determining the number of rows and columns of a first initial attention matrix and the number of rows and columns of a second initial attention matrix according to the number of rows of the sample matrix; Initialize the weight values ​​of the first initial attention matrix to obtain the corresponding forward attention weight matrix, and initialize the weight values ​​of the second initial attention matrix to obtain the corresponding reverse attention weight matrix.

5. The method according to claim 4, characterized in that The constructing of a bidirectional attention weight matrix without self-information based on the forward attention weight matrix and the reverse attention weight matrix includes: Splicing the forward attention weight matrix and the reverse attention weight matrix to obtain a spliced ​​matrix; the forward attention weight matrix and the reverse attention weight matrix are respectively divided into a first part and a second part along their own diagonals; The second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix are masked to obtain a bidirectional attention weight matrix without self-information.

6. The method according to claim 5, characterized in that The step of masking the second part of the forward attention weight matrix and the first part of the reverse attention weight matrix in the splicing matrix to obtain a bidirectional attention weight matrix without self-information includes: The first mask template in the splicing matrix is ​​used to cover the second part of the forward attention weight matrix, and the second mask template is used to cover the first part of the reverse attention weight matrix to obtain a bidirectional attention weight matrix without its own information.

7. A model application method, characterized in that: include: Obtain the data to be predicted; The model trained by the model training method according to any one of claims 1 to 6 is used to predict the data to be predicted to obtain a prediction result.

8. A model training device, characterized in that: The device comprises: Sample acquisition module, used to obtain training sample data; A first construction module is used to construct corresponding forward attention weight matrices and reverse attention weight matrices according to the training sample data, wherein the forward attention weight matrix is ​​used to predict the next adjacent element of each element in the training sample data, and the reverse attention weight matrix is ​​used to predict the previous adjacent element of each element in the training sample data; A second construction module is used to construct a bidirectional attention weight matrix without self-information based on the forward attention weight matrix and the reverse attention weight matrix; A model training module is used to process the training sample data according to the forward attention weight matrix to obtain a first loss; process the training sample data according to the reverse attention weight matrix to obtain a second loss; process the training sample data according to the bidirectional attention weight matrix to obtain a third loss; determine the total loss based on the first loss, the second loss and the third loss, and train the preset model based on the total loss.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images